Skip to content

feat(web): fetch /browse data client-side and disallow crawlers via robots.txt#1426

Merged
brendan-kellam merged 5 commits into
mainfrom
brendan/browse-csr
Jul 8, 2026
Merged

feat(web): fetch /browse data client-side and disallow crawlers via robots.txt#1426
brendan-kellam merged 5 commits into
mainfrom
brendan/browse-csr

Conversation

@brendan-kellam

@brendan-kellam brendan-kellam commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Problem

app.sourcebot.dev was repeatedly killed by its liveness probe (6 restarts in 24h). Investigation traced the cause to /browse pages: blob/tree/commit panels fetched git data (file sources, folder contents, full commit diffs) in server components and passed it as props across the client-component boundary. React flight-encodes those props into the document on the main thread — a single large commit page measured 38MB of RSC payload (~8s of encoding), and a binary blob produced a 62MB document. Crawler traffic (GPTBot, Applebot, meta-externalagent, AhrefsBot — ~97% of all server processing time) walked the unbounded @<sha> URL space, stacking these renders until the event loop starved and health probes timed out.

Changes

  1. robots.txt (app/robots.ts): disallow all crawling, with an allowlist for link-preview fetchers (Slackbot, Twitterbot, LinkedInBot, Discordbot, facebookexternalhit, TelegramBot, WhatsApp) so shared links — e.g. chat pages — keep their OpenGraph unfurls. Applebot/GPTBot/meta-externalagent are deliberately not allowlisted (search/AI-training crawlers, not preview fetchers).

  2. Client-side data fetching for /browse panels: CodePreviewPanel, TreePreviewPanel, FullCommitDiffPanel, and FocusedCommitDiffPanel now render a shell and fetch via JSON API routes with react-query (per repo convention). Server components only fetch repoInfo for the path header.

    • New /api/folder_contents route (internal, not registered in the public API) wrapping the existing getFolderContents.
    • New client fetchers: getCommit, getDiff, getFolderContents, getFileBlame.
    • Shared useCommitDiff hook preserving the initial-commit empty-tree fallback.
    • CommitsPanel left server-side intentionally (bounded at 35 commits/page).

Documents drop to the ~70KB app shell regardless of content size, JSON serialization replaces flight encoding (~50-100x less main-thread time for large payloads), and JS-less crawlers never trigger data fetches at all.

Verification

Exercised in an isolated dev environment against an indexed repo: blob, tree, commit, blame-mode, and focused-diff pages all render correctly via client fetching; blob documents contain zero file content and commit documents zero hunk payloads (previously embedded); all API routes return 200; robots.txt serves the intended rules. Typecheck clean for all touched files.

Follow-ups (not in this PR)

  • Payload caps on the JSON APIs themselves (/api/diff on a mega-commit still returns MBs of JSON).
  • Liveness probe timeout bump + event-loop load shedding (infra/chart).

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Browse pages now fetch file sources, folder contents, diffs, and blame in the browser via API routes for a smoother experience.
    • Commit and tree preview panels were updated to render client-side loading/error states and interactive UI.
  • Bug Fixes
    • Improved diff handling for initial commits (files show as added by default).
    • More reliable matching for file renames and deletions when viewing diffs.
  • Chores
    • Added a robots.txt route that blocks crawlers by default while allowing common link-preview bots to keep OpenGraph previews working.

brendan-kellam and others added 2 commits July 7, 2026 18:49
…w bots

Sourcebot exposes an unbounded URL space (every file x revision x commit)
and crawler traffic (GPTBot, Applebot, meta-externalagent, AhrefsBot)
generates heavy SSR load. Deny all crawling while keeping unfurl bots
(Slack, X, LinkedIn, etc.) allowed so shared links keep their OpenGraph
previews.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…C props

Blob, tree, and commit-diff panels previously fetched git data (file
source, folder contents, full commit diffs) in server components and
passed it as props across the client boundary. React flight-encodes
those props into the document at seconds of main-thread time for large
payloads (a large commit page measured 38MB / ~8s of encoding), which
starves the event loop and health probes under crawler load.

The panels now render a shell and fetch via existing JSON API routes
with react-query (per repo convention), plus a new /api/folder_contents
route. Documents drop to the ~70KB shell regardless of content size,
and JS-less crawlers never trigger the data fetch at all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5786f4f6-5cd0-4eb5-a948-d2d82fccae7d

📥 Commits

Reviewing files that changed from the base of the PR and between 98aa8a7 and badd362.

📒 Files selected for processing (1)
  • packages/web/src/app/(app)/browse/[...path]/components/codePreviewPanel/codePreviewPanelClient.tsx

Walkthrough

Browse preview and commit-diff panels now fetch only repository metadata on the server and load file, folder, commit, diff, and blame data in client components. New client API helpers, a folder contents API route, changelog notes, and a robots policy accompany the refactor.

Changes

Browse panel client-side refactor

Layer / File(s) Summary
Client API wrappers and folder_contents route
packages/web/src/app/api/(client)/client.ts, packages/web/src/app/api/(server)/folder_contents/route.ts
Adds getCommit, getDiff, getFolderContents, and getFileBlame client helpers, plus a validated GET route for folder contents.
Code preview panel client migration
packages/web/src/app/(app)/browse/[...path]/components/codePreviewPanel/codePreviewPanel.tsx, packages/web/src/app/(app)/browse/[...path]/components/codePreviewPanel/codePreviewPanelClient.tsx
CodePreviewPanel now fetches only repo info and delegates to CodePreviewPanelClient, which fetches file source and blame, computes file metadata, and renders preview and header controls.
Tree preview panel client migration
packages/web/src/app/(app)/browse/[...path]/components/treePreviewPanel/treePreviewPanel.tsx, packages/web/src/app/(app)/browse/[...path]/components/treePreviewPanel/treePreviewPanelClient.tsx
TreePreviewPanel now fetches only repo info and delegates to TreePreviewPanelClient, which fetches folder contents and renders the tree header and items.
Commit diff panels client migration
packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanel.tsx, packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanelClient.tsx, packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/fullCommitDiffPanel.tsx, packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/useCommitDiff.ts
FocusedCommitDiffPanel and FullCommitDiffPanel now delegate commit and diff loading to useCommitDiff, and FocusedCommitDiffPanelClient renders the matching file diff, metadata, and viewer.
Changelog updates
CHANGELOG.md
Adds entries for the browse-page client-side fetch refactor and the robots route.

Estimated code review effort: 4 (Complex) | ~60 minutes

Robots.txt Crawler Policy

Layer / File(s) Summary
Robots configuration
packages/web/src/app/robots.ts
Introduces robots() returning MetadataRoute.Robots with an allowlist for preview bots and a catch-all disallow rule.

Sequence Diagram(s)

sequenceDiagram
  participant CodePreviewPanel
  participant CodePreviewPanelClient
  participant ClientAPI
  CodePreviewPanel->>ClientAPI: getRepoInfoByName(repoName)
  CodePreviewPanel->>CodePreviewPanelClient: render with repo, path, revisionName
  CodePreviewPanelClient->>ClientAPI: getFileSource(...)
  CodePreviewPanelClient->>ClientAPI: getFileBlame(...)
  ClientAPI-->>CodePreviewPanelClient: source, blame data
  CodePreviewPanelClient-->>CodePreviewPanelClient: render PureCodePreviewPanel
Loading
sequenceDiagram
  participant FocusedCommitDiffPanel
  participant FocusedCommitDiffPanelClient
  participant useCommitDiff
  participant ClientAPI
  FocusedCommitDiffPanel->>ClientAPI: getRepoInfoByName(repoName)
  FocusedCommitDiffPanel->>FocusedCommitDiffPanelClient: render with repo, commitSha, path
  FocusedCommitDiffPanelClient->>useCommitDiff: useCommitDiff(repoName, commitSha, path)
  useCommitDiff->>ClientAPI: getCommit(...)
  useCommitDiff->>ClientAPI: getDiff(...)
  ClientAPI-->>useCommitDiff: commit, diff
  useCommitDiff-->>FocusedCommitDiffPanelClient: commit, diff
  FocusedCommitDiffPanelClient-->>FocusedCommitDiffPanelClient: render file header and LightweightDiffViewer
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly captures the two main changes: client-side /browse data fetching and crawler blocking via robots.txt.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch brendan/browse-csr

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
packages/web/src/app/api/(client)/client.ts (1)

173-240: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Optional: Extract a shared fetch helper to reduce duplication.

The four new functions (getCommit, getDiff, getFolderContents, getFileBlame) share the same structure: URL construction, param iteration, fetch with the X-Sourcebot-Client-Source header, JSON parsing, and type casting. A small shared helper would centralize the pattern and make future additions or header changes a one-line edit.

♻️ Suggested shared helper
+async function fetchJson<T>(endpoint: string, queryParams: Record<string, string | undefined>): Promise<T | ServiceError> {
+    const url = new URL(endpoint, window.location.origin);
+    for (const [key, value] of Object.entries(queryParams)) {
+        if (value !== undefined) {
+            url.searchParams.set(key, value);
+        }
+    }
+    const result = await fetch(url, {
+        method: "GET",
+        headers: {
+            "X-Sourcebot-Client-Source": "sourcebot-web-client",
+        },
+    }).then(response => response.json());
+    return result as T | ServiceError;
+}
+
-export const getCommit = async (queryParams: { repo: string; ref: string }): Promise<CommitDetail | ServiceError> => {
-    const url = new URL("/api/commit", window.location.origin);
-    for (const [key, value] of Object.entries(queryParams)) {
-        url.searchParams.set(key, value);
-    }
-
-    const result = await fetch(url, {
-        method: "GET",
-        headers: {
-            "X-Sourcebot-Client-Source": "sourcebot-web-client",
-        },
-    }).then(response => response.json());
-
-    return result as CommitDetail | ServiceError;
-}
+export const getCommit = (queryParams: { repo: string; ref: string }) =>
+    fetchJson<CommitDetail>("/api/commit", queryParams);

The same simplification applies to getDiff, getFolderContents, and getFileBlame. The undefined filter in the helper covers both required and optional param types uniformly.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/web/src/app/api/`(client)/client.ts around lines 173 - 240, The four
API client functions repeat the same URL-building and fetch/json/header logic,
so extract that common flow into a shared helper in client.ts and have
getCommit, getDiff, getFolderContents, and getFileBlame delegate to it. Keep the
helper responsible for appending query params, setting the
X-Sourcebot-Client-Source header, performing fetch, and returning parsed JSON so
future header or request changes only need one update.
packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanelClient.tsx (1)

42-57: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Optional: extract the shared pending/error UI.

These pending and error blocks are duplicated verbatim in fullCommitDiffPanel.tsx (lines 25-40). Consider a small shared wrapper (e.g. <CommitDiffState isPending error>) to avoid divergence as the panels evolve.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/web/src/app/`(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanelClient.tsx
around lines 42 - 57, The pending and error UI in focusedCommitDiffPanelClient
is duplicated in fullCommitDiffPanel, so extract the shared loading/error
rendering into a reusable component or wrapper (for example, a CommitDiffState
in the shared commit diff panel area) and use it from both panels. Keep the
existing behavior of the isPending and error/data checks, but centralize the
Loader2/loading text and error message markup so both
focusedCommitDiffPanelClient and fullCommitDiffPanel stay in sync as they
evolve.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@packages/web/src/app/`(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanelClient.tsx:
- Around line 42-57: The pending and error UI in focusedCommitDiffPanelClient is
duplicated in fullCommitDiffPanel, so extract the shared loading/error rendering
into a reusable component or wrapper (for example, a CommitDiffState in the
shared commit diff panel area) and use it from both panels. Keep the existing
behavior of the isPending and error/data checks, but centralize the
Loader2/loading text and error message markup so both
focusedCommitDiffPanelClient and fullCommitDiffPanel stay in sync as they
evolve.

In `@packages/web/src/app/api/`(client)/client.ts:
- Around line 173-240: The four API client functions repeat the same
URL-building and fetch/json/header logic, so extract that common flow into a
shared helper in client.ts and have getCommit, getDiff, getFolderContents, and
getFileBlame delegate to it. Keep the helper responsible for appending query
params, setting the X-Sourcebot-Client-Source header, performing fetch, and
returning parsed JSON so future header or request changes only need one update.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c02f0a2b-b454-4983-964c-cb35fa6d5583

📥 Commits

Reviewing files that changed from the base of the PR and between 94b2af6 and 80ee7fa.

📒 Files selected for processing (12)
  • CHANGELOG.md
  • packages/web/src/app/(app)/browse/[...path]/components/codePreviewPanel/codePreviewPanel.tsx
  • packages/web/src/app/(app)/browse/[...path]/components/codePreviewPanel/codePreviewPanelClient.tsx
  • packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanel.tsx
  • packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanelClient.tsx
  • packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/fullCommitDiffPanel.tsx
  • packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/useCommitDiff.ts
  • packages/web/src/app/(app)/browse/[...path]/components/treePreviewPanel/treePreviewPanel.tsx
  • packages/web/src/app/(app)/browse/[...path]/components/treePreviewPanel/treePreviewPanelClient.tsx
  • packages/web/src/app/api/(client)/client.ts
  • packages/web/src/app/api/(server)/folder_contents/route.ts
  • packages/web/src/app/robots.ts

brendan-kellam and others added 2 commits July 7, 2026 20:00
Render the path header, separator, and file toolbar (blame toggle + age
legend) immediately from props, and scope the loading spinner / error to
the body below. The line-count/size stat shows a skeleton while the file
source is pending, then the real value once loaded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@brendan-kellam brendan-kellam merged commit 2afc265 into main Jul 8, 2026
10 checks passed
@brendan-kellam brendan-kellam deleted the brendan/browse-csr branch July 8, 2026 03:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant